Skip to content

[SPARK-29112][YARN] Expose more details when ApplicationMaster reporter faces a fatal exception #25810

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
LantaoJin wants to merge 1 commit into from
Closed

[SPARK-29112][YARN] Expose more details when ApplicationMaster reporter faces a fatal exception #25810

LantaoJin wants to merge 1 commit into from

Conversation

@LantaoJin
Copy link
Contributor

@LantaoJin LantaoJin commented Sep 17, 2019

What changes were proposed in this pull request?

In ApplicationMaster.Reporter thread, fatal exception information is swallowed. It's better to expose it.
We found our thrift server was shutdown due to a fatal exception but no useful information from log.

19/09/16 06:59:54,498 INFO [Reporter] yarn.ApplicationMaster:54 : Final app status: FAILED, exitCode: 12, (reason: Exception was thrown 1 time(s) from Reporter thread.)
19/09/16 06:59:54,500 ERROR [Driver] thriftserver.HiveThriftServer2:91 : Error starting HiveThriftServer2
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:160)
at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:708)

Does this PR introduce any user-facing change?

No

How was this patch tested?

Manual test

@LantaoJin
Copy link
Contributor Author

LantaoJin commented Sep 17, 2019 •
edited
Loading

@jerryshao Could you have a time to review?

@wangyum
Copy link
Member

wangyum commented Sep 17, 2019

ok to test

@SparkQA
Copy link

SparkQA commented Sep 17, 2019

Test build #110741 has finished for PR 25810 at commit 8b14afe.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member

wangyum commented Sep 17, 2019

add to whitelist

@LantaoJin
Copy link
Contributor Author

LantaoJin commented Sep 17, 2019

retest this please

@SparkQA
Copy link

SparkQA commented Sep 17, 2019

Test build #110766 has finished for PR 25810 at commit 8b14afe.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 17, 2019

Test build #110763 has finished for PR 25810 at commit 8b14afe.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

jerryshao
jerryshao approved these changes Sep 18, 2019
View reviewed changes
Copy link
Contributor

@jerryshao jerryshao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@jerryshao
Copy link
Contributor

jerryshao commented Sep 18, 2019

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Sep 18, 2019

Test build #110856 has finished for PR 25810 at commit 8b14afe.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jerryshao
Copy link
Contributor

jerryshao commented Sep 18, 2019

LGTM, merging to master branch.

@jerryshao jerryshao closed this in 0b6775e Sep 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jerryshao jerryshao jerryshao approved these changes

Assignees

No one assigned

Labels

YARN

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@LantaoJin @wangyum @SparkQA @jerryshao @dongjoon-hyun